Purpose

Linked issue: close #452

A data-evolution partial update rewrites only the touched rows of a blob column and records every untouched row as a placeholder entry (bin_length -2, no data bytes). This PR makes the whole chain understand such entries:

• Blob format, write side: the placeholder protocol is opt-in via the internal format option blob.internal.write-placeholder (BlobDefs::kWritePlaceholderKey, false by default), enabled only for data-evolution partial updates, i.e. blob-only column writes of a table with data evolution enabled. Under the protocol, a value exactly equal to the internal reserved sentinel _PAIMON_BLOB_PLACEHOLDER (BlobDefs::kPlaceholderSentinel, exposed as BlobDefs::PlaceholderSentinelView()) is persisted as a bin_length -2 entry without payload bytes; any other value is stored verbatim. Outside the protocol the writer never interprets values at all.
• Blob format, read side: the reader fails on placeholder entries by default; the internal option blob.internal.emit-placeholder-sentinel (BlobDefs::kEmitPlaceholderSentinelKey, false by default, set only by the fallback read path) switches it to emit the sentinel bytes for -2 entries, so after the batch has passed through schema-mapping readers, the fallback merge can identify placeholders by exact byte equality. There is no escaping anywhere in the chain: inside those two internal channels a user blob whose bytes equal the sentinel is indistinguishable from a placeholder and would be persisted as one. The sentinel is distinctive enough that this collision is accepted as negligibly improbable (documented on BlobDefs::kPlaceholderSentinel). Sentinel bytes are never stored in blob files.
• BlobBunch: keeps all blob files of a bunch, where files sharing a max_sequence_number form one non-overlapping layer; overlaps across layers are the expected shape of partial updates. When every file shares a single max_sequence_number, SequentialReadOptimize() keeps the existing concat fast path.
• BlobFallbackBatchReader (new, aligned with Java's BlobFallbackRecordReader / AllPlaceholdersRecordReader): merges the layers of one blob bunch ordered by max_sequence_number descending; row-id ranges a layer does not cover are padded with placeholder gap segments so all layers step in lockstep; each output row takes the newest non-placeholder layer, and an explicitly written null wins over older layers. A row that is a placeholder in every layer degrades to a null blob while keeping its _ROW_ID and reporting -1 as its _SEQUENCE_NUMBER; resolved rows report their layer's sequence number. Row-range pushdown is honored per layer, including inside gaps. Unlike Java's ForceSingleBatchReader, the layers are stepped in a read_batch_size window rather than materialized as a single batch.
• BlobFileBatchReader: GetPreviousBatchFileRowId now stays usable when a selection bitmap removed rows, mapping batch positions back to original file row indexes through target_blob_row_indexes_ — required by CompleteRowTrackingFieldsBatchReader to synthesize _ROW_ID under row-range pushdown.
• CompleteRowTrackingFieldsBatchReader: takes an explicit optional file_field_names (from DataFileMeta::write_cols) declaring the physical fields of a format without a self-describing file schema (blob); self-describing formats keep querying GetFileSchema(). Unrelated NotImplemented errors stay visible instead of being treated as an empty schema.
• DataEvolutionSplitRead: builds the fallback reader when a blob bunch spans multiple sequence layers, and passes the internal format options through AbstractSplitRead::CreateRawFileReaders (explicit extra_format_options parameter, no default arguments).
• DataEvolutionFileReader: Create accepts a single inner reader — a blob-only write collapses every layer of a merge split into one blob bunch, and the degenerate union still maps the bunch's fields into read schema order and null fills unmatched read fields. Reader offsets are now validated against the reader count instead of requiring at least two readers.
• DataEvolutionBatchScan: WrapToIndexedSplits derived a split's row id range from its first and last file only. That bounding range relies on an invariant that does not hold — the row-id ranges of the files in a split may be unordered, discontiguous or overlapping, since DataEvolutionSplitGenerator can bin-pack disjoint groups into one split — so it both admitted row ids no file of the split covers and, for a partial-update blob layer covering a strict subrange, truncated the range and silently dropped requested row ranges at scan time. The index is now intersected with each file's row-id range separately and the results are sorted and merged (Range::SortAndMergeOverlap(..., adjacent=true)), aligning with Java's DataEvolutionBatchScan.wrap(). WrapToIndexedSplits becomes static and visible for testing.

Tests

• blob_format_writer_test.cpp: placeholder golden bytes aligned with the Java writer, strict vs placeholder-aware read modes (including descriptor mode and selection bitmaps), and sentinel-equal / sentinel-prefixed user bytes stored verbatim (TestWritePlaceholderGoldenBytes, TestReadPlaceholderStrictAndAwareModes, TestReadPlaceholderWithSelectionBitmap, TestSentinelBytesVerbatimWithoutPlaceholderMode, TestSentinelPrefixedValueVerbatimInPlaceholderMode).
• blob_file_batch_reader_test.cpp: batch positions map back to original file row indexes under a selection bitmap (TestRowNumbersWithSelectionBitmap).
• complete_row_tracking_fields_reader_test.cpp: declared physical fields take precedence over the inner reader's self-described schema (TestSetReadSchemaWithDeclaredFileFields); existing cases keep exercising the self-describing branch.
• blob_fallback_batch_reader_test.cpp (new): fallback merge across layers under batch-size sweeps — gaps (leading/middle/trailing, disjoint ranges), all-placeholder rows degrading to null, null-wins semantics, sentinel-prefixed pass-through, three layers, row-tracking projections including the all-placeholder -1 semantics, and misaligned-group / creation failures.
• data_evolution_batch_scan_test.cpp (new): WrapToIndexedSplits over a split whose files have unordered and discontiguous row-id ranges (mirroring Java's testWrapToIndexSplitsWithUnorderedAndDiscontiguousDataFiles), row ids falling into the gaps between file ranges being excluded, and the no-intersection failure (TestWrapToIndexedSplitsWithUnorderedAndDiscontiguousDataFiles, TestWrapToIndexedSplitsExcludesRowIdsInFileRangeGaps, TestWrapToIndexedSplitsWithoutIntersection).
• data_evolution_split_read_test.cpp: BlobBunch keeps all sequence layers, rejects overlaps within one layer, and reports RowCount / SequentialReadOptimize accordingly.
• data_evolution_file_reader_test.cpp: creation rejects empty readers and reader offsets referencing a missing reader, replacing the old at-least-two-readers requirement (TestInvalid).
• blob_table_inte_test.cpp (IT): end-to-end partial-update reads — placeholder fallback with null overwrite and a blob_as_descriptor variant, multi-layer and compacted-layer layouts (mirroring Java's BlobUpdateTest.testReadCompactedBlobSequenceGroups) with per-row and gap-crossing row-range reads, row-range pushdown over updated and untouched rows, a subrange sequence layer under the row-tracking projection with row-range pushdown, an all-placeholder row keeping its _ROW_ID with _SEQUENCE_NUMBER -1, and sentinel-equal / sentinel-prefixed user blobs passing through unchanged (TestDataEvolutionBlobPartialUpdate*, TestBlobValueEqualToPlaceholderSentinelBytes). The row-tracking tests write only the blob column, so the split contains only blob files and the blob bunch itself carries the row-tracking fields.

API and Format

• No changes under include/paimon.
• Blob file format: bin_length -2 marks a placeholder entry occupying no file space, aligned with Java's BlobFormatWriter.PLACE_HOLDER_LENGTH. Readers without placeholder support already reject such entries with an explicit error, and files without placeholders are unaffected.
• The in-repo blob format description (blob_file_batch_reader.h) now documents both special index values: -1 (null) and -2 (placeholder).
• _PAIMON_BLOB_PLACEHOLDER is an internal reserved value only inside the two internal placeholder channels; normal writes and reads never interpret it.
• Two new internal (non user-facing) format options, both false by default: blob.internal.write-placeholder (set only for data-evolution blob-only column writes) and blob.internal.emit-placeholder-sentinel (set only by the data-evolution fallback read path).

Documentation

Internal read/write behavior of data-evolution partial updates; no user-facing documentation change.

Generative AI tooling

Generated-by: Claude Code (Claude Fable 5)

🤖 Generated with Claude Code